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Abstract 

Over recent years Value of Information analysis has become more widespread in health-economic 
evaluations, specifically as a tool to perform Probabilistic Sensitivity Analysis. This is largely due 
to methodological advancements allowing for the fast computation of a typical summary known as 
the Expected Value of Partial Perfect Information (EVPPI). A recent review discussed some estim¬ 
ations method for calculating the EVPPI but as the research has been active over the intervening 
years this review does not discuss some key estimation methods. Therefore, this paper presents 
a comprehensive review of these new methods. We begin by providing the technical details of 
these computation methods. We then present a case study in order to compare the estimation 
performance of these new methods. We conclude that the most recent development based on non- 
parametric regression offers the best method for calculating the EVPPI efficiently. This means 
that the EVPPI can now be used practically in health economic evaluations, especially as all the 
methods are developed in parallel with R functions to aid practitioners. 

Keywords: Probabilistic Sensitivity Analysis; Value of Information; Computation Methods; 
EVPPI □ 

1 Introduction 

Health economic models are used to predict the costs and health effects of competing healthcare 
technology decision options. Model input parameters are informed by real-world study evidence 
and are therefore rarely known with certainty. The typical method for assessing input uncertainty 
is Probabilistic Sensitivity Analysis (PSA), i.e. Monte Carlo or bootstrap sampling from the input 
parameter probability distribution coupled with model evaluation in order to generate samples of 
predicted costs and effects. PSA is a fundamental component of any health economic evaluation 
and is recommended by Health Technology Assessment (HTA) bodies such as the National Institute 
for Health and Care Excellence (NICE) in the UK [12], across other countries in Europe m and 
beyond mm- 

A fully decision-theoretic approach to PSA is based on the analysis of the value of information 
(voi) m, an increasingly popular method in health economic evaluation [I] Ei El llOl 111! 1161 - 
i!8l l22t 128] . A Voi analysis quantifies the expected value of obtaining perfect information about 
the underlying model parameters. Because it is usually impossible to obtain perfect information, 
this analysis places an upper bound on the cost of additional research aimed at reducing this 
uncertainty. 

The main advantage of the analysis of the Voi is that it directly addresses the potential implic¬ 
ations of current uncertainty, not only in terms of the likelihood of modifying the current decision 
in light of new and more definitive evidence, but also in terms of the opportunity cost of the 
incorrect decision. If this cost is low, there is little value in agonising about the decision, even 
for a low probability of cost-effectiveness, as the implicit penalty is negligible if the decision turns 
to be wrong in the face of new evidence. Therefore, the probability of cost-effectiveness alone 
can dramatically overstate decision sensitivity. For this reason, it has been advocated that Voi 
measures should also be presented when representing decision uncertainty [2J 21 ll6l [24] . 

1 Financial support for this study was provided in part by grants from EPSRC [Anna Heath] and Mapi [Dr. Gianluca 
Baio], The funding agreement ensured the authors independence in designing the study, interpreting the data, writing, 
and publishing the report. 
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Despite the useful features of a Vol analysis, its uptake in health economic evaluation has been 
slow. Vol analysis has been hindered by two factors in the main. First, the interpretation of the 
expected value of information is not straightforward as Vol measures are theoretically unbounded, 
making interpretation more challenging. 

Secondly, and perhaps more importantly, Vol measurements can be computationally costly 
unless restrictive assumptions are made [33]. Practically, it is typically easy to calculate numerically 
(but not analytically) the expected value of learning all the model parameters perfectly. This is 
known as the (overall) Expected Value of Perfect Information (EVPI). However, this has often 
little practical use, as it will be rarely possible to learn all the model parameters at once in a 
new study. Thus, the decision-maker is usually interested in the expected value of information 
about subsets of parameters. This will indicate which parameters are driving decision uncertainty, 
and thus where future research may add value. This subset analysis is called the Expected Value 
of Perfect Partial Information (EVPPI), and it is this analysis that is computationally costly. 
Therefore, significant effort has been invested in developing more efficient methods for computing 

EVPPI 0 El H31 US H71 [3T] EU ■ 

A relatively recent review El discussed some of these methods. However, research in this 
area has been active over the last few years. We present and evaluate these recent methodological 
advances, concluding that the most recent developments made in [321 offer the best method for 
approximating the EVPPI. 

We begin by formalising the EVPPI calculation and briefly discussing the methods from the 
previous review. We then present two methods for single-parameter EVPPI before discussing the 
most recent developments based on non-parametric regression. Then, we compare the performance 
of these methods using a fictional but realistic example of economic modelling. From our analysis, 
it appears that all methods produce comparable results and have similar standard errors and 
computational time. Finally, we present some discussion to suggest that, although they are still 
affected by some computational limitations, non-parametric regression methods offer the most 
suitable EVPPI approximation method. 


2 Notation and basic concepts 

Health economic decision making aims at determining the optimal intervention in terms of costs 
and health effects, among a number T of options, often set to T = 2. In general we can consider the 
case where a larger number of interventions is available, although in practice it is uncommon that 
T > 5. Usually, we consider t — 0 as the standard intervention currently available for treatment 
of a specific condition and t = 1,..., T as a (set of) new option(s) being assessed. 

Each alternative t = 0, ..., T is valued by means of a utility function, typically defined in terms 
of the monetary net benefit 

nb t = ke t - c t . 

Here, (et,Ct) is the health economic response, typically subject to individual variability expressed 
by a joint probability distribution p(e, c j 0), indexed by a set of relevant parameters 0, while k is 
the willingness to pay. 

In a full Bayesian setting, a complete ranking of the alternatives is obtained by computing 
the overall expectation of this utility function over both individual variability and parameter 
uncertainty 

MBt = kE[e t \ -E[ct], 

i.e. the expectation here is taken with respect to the joint distribution p(e, c, 0) = p{e, c \ 9)p(0). 
The option t associated with the maximum overall expected utility MB* = maxtMBt is deemed 
to be the most “cost-effective”, given current evidence. 

However, when performing PSA, we consider the expected utility taken with respect to indi¬ 
vidual variability only (i.e. as a function of the parameters 0) 

NB t (0) = kE[e t | 0] -E[ct | 0], 

which, in line with [2], we term the “known-distribution” net benefit. In this case, the expectation 
is taken with respect to the conditional distribution p(e, c | 0). Thus, while MBt is a deterministic 
quantity, NBt(0) is a random variable (randomness being induced by uncertainty in the model 
parameters 0) and E[NB t (0)] = MBt. 

The vector of parameters can in general be split in two components 0 = (</>, ip), where <p is the 
sub-vector of parameters of interest (i.e. those that could be investigated further) and ip are the 
remaining “nuisance” parameters. To calculate the EVPPI, we compute a weighted average of the 
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net benefit for the optimal decision at every point in the support of <j> after having marginalised 
out the uncertainty due to ip. 


EVPPI = E 0 


max E ^,|0 [NB t (0)] 


maxEe [NB t (0)], 


(1) 


where maxtE^^ [NB t (0)] is the value of learning <fi with no uncertainty. Of course, this is only 
an idealised state of nature, as we will hardly even be in the position of completely eliminating 
the uncertainty on </>; thus, we then average over its current probability distribution. 


3 A review of available methods for the computation 
of the EVPPI in health economic evaluations 

It is rarely possible to calculate the EVPPI analytically as the known distribution net benefit is 
often a complicated function of the underlying parameters. Additionally, it is generally challenging 
to calculate analytically the expectation of a maximum, as is required in the first term of 0 - For 
this reason, a wealth of papers have been dedicated to finding accurate approximation methods 
for the EVPPI. 


3.1 EVPPI approximation methods requiring additional sampling 
or underlying assumptions 

The most recent methodological review of these methods m presented four approximation pro¬ 
cedures for the EVPPI. Two of these methods are computationally intensive and derive directly 
from the construction of the first term in 0 - The other two methods can only be used when the 
known-distribution net benefit function fulfils certain conditions, generally involving some form of 
linearity in the parameter sets and/or simplifying distributional assumptions. These methods are 
considerably faster in terms of computational time. 

The most important computationally intensive method is a two-step Monte Carlo procedure 
that was first formalised in [ 6 ]. For this method, Monte Carlo sampling is used to approximate both 
the inner and outer expectations in the first term of (Tj). The conditional expectation E^,| 0 [NBt( 0 )] 
must be calculated using a nested simulation as it is dependent on <p. Therefore, the main limitation 
of this method is the immense computation power needed to calculate an accurate estimate for 
the EVPPI. Although theoretically Monte Carlo simulation can give any desired accuracy, the 
number of nested simulations necessary to produce accurate estimates would generally require an 
infeasibly long computation time, making this method impractical, despite theoretically offering 
good estimation properties. The second expensive algorithm uses numerical integration to reduce 
the number of nested samples needed. However, it does not offer a significant computational 
saving when compared to the two-step Monte Carlo method and is impractical for high-dimensional 
subsets of cp. 

The faster, restrictive methods for EVPPI calculation include the unit loss integral approxim¬ 
ation, which assumes the parameters of interest <f> are approximately normal and uses this approx¬ 
imation to derive analytical results. The second restrictive approximation was first developed in 
|16] under the assumptions that the net benefit is a linear function in the “nuisance” parameters 
ip and that these are independent of the relevant parameters <p (although these conditions can be 
slightly relaxed 03 [20). These assumptions ensure that it is possible to calculate the EVPPI using 
the samples from the parameters distribution, which are already available from PSA. However, 
this approximation deteriorates if the independence assumption is not valid. 

More recently, Madan et al. [53] explored some re-parametrisations and approximations of the 
net benefit such that this method can be used when its original formulation does not conform to 
these restrictive conditions. For example, if the net benefit is non-linear in ip but ip and cp are 
independent, it would then simply be possible to re-parametrise the problem so that NB t is linear 
in a new parameter set ip' = f(ip). 

All of these methods rely on additional information about the model — either to perform 
more sampling or to check whether it conforms to certain conditions — making EVPPI calcula¬ 
tion challenging, both methodologically and computationally. However, since the review of 113) . 
significant developments have been made allowing the EVPPI to be calculated quickly using the 
data already available as part of a standard PSA procedure with no knowledge of the underlying 
model structure. 
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3.2 Approximations for Single Parameter EVPPI 

Recently, two methods have been specifically developed to approximate the EVPPI for a single 
parameter, cj>, which rely on the same underlying ideas: i) provided the optimal decision remains 
constant, the first term for the EVPPI can be approximated by the average known-distribution 
net benefit value within this set; and ii) if a treatment option is optimal at one point in the 
parameter set, it is still optimal for parameter values “close to” that original point. For a set of S 
simulated parameter values (9 1 ,..., 6 s ), it is possible, therefore, to approximate the first term in 
<[T|) provided there is some assurance that the optimal decision remains constant within the sets of 
interest. 

3.2.1 Strong and Oakley 

The first method m is based on the idea that if the parameter space is split into “small” subsets, 
the optimal decision is unlikely to change within each of these subsets, as it is reasonable to assume 
it is locally constant. 

Thus, to calculate the EVPPI, it is necessary to determine subsets of the simulated values of 
NBt(0) for which the simulated values of (j> are similar. Practically, this is achieved by reordering 
the observed known-distribution net benefit values so that they have to same order as the simulated 
values for (j) — note that since it is assumed that (j) is a scalar, ordering is trivial. This list of 
ordered values is then split into M small sub-lists of length L = jj. Within each sub-list the 
average known-distribution net benefit is calculated for each treatment option and the maximum 
within each subset is used as an estimate for the first term in equation 0. in correspondence of 
the simulated value for 0. 

Therefore, the following strategy can be used: 

1. Sample S values from the distribution of the parameters 9\ we indicate these as 0 1 ,..., 0 s . 

2. At each iteration s = 1,..., S, use the simulated parameter vector 9 a to compute the estim¬ 
ated known-distribution net benefit NBt(0 s ) for each treatment option t. 

3. Sort the simulated values of the parameter of interest in ascending order — for simplicity, 
we write the sorted vector as ip 1 ,... ,p s , where min (f> = p 1 < ... < p s = max <fi. 

4. Re-order the estimated known-distribution net benefits as NB t (ip 1 ),. .., NBt((p s ), where 
NBt(i^ s ) is the net benefit corresponding to the s —th ordered simulated value of 4>. 

5. Split the ordered list of known-distribution net benefits into m = 1,..., M sub-lists C m and 
compute the average in each of the sub-lists and for each treatment t to obtain 

NB™(0) = i NB *(^)- 
jeCm 

M 

6. Compute EVPPI = ~ ^ maxNBT(0). 

m=1 

The approximation given by this method is highly sensitive to the value of M. If M is large, 
the EVPPI is close to the overall EVPI estimate. At the other end of the scale, if M is small, 
the EVPPI is approximately 0. Strong and Oakley m use a normal approximation to estimate 
the upward bias of the EVPPI calculated using this method. They suggest choosing M to be the 
largest number of subsets such that the upward bias falls below a pre-specified threshold value, 
say 0.1. In this manner, the upward bias of the EVPPI estimate is controlled but, hopefully, the 
downward bias, present for small values of M, is avoided. 

3.2.2 Sadatsafavi et al. 

This second method for single parameter EVPPI m can be thought of as an extension to the 
previous one, although both were developed concurrently. In this case, the ordered list 


/ NBo(^) • 

•• nb 0 (v? s ; 

NB(fl) = ' 

• ■ nbi(v 3 s ; 

\ NBt^ 1 ) • 

NBt(v2 S 


is split in an informed manner. The algorithm searches for the point (s) at which the parameter of 
interest is directly responsible for a change in the optimal decision. Although, in general, variations 
in a parameter may induce a change in the optimal decision a large number M of times, in practice 


4 


a single parameter is unlikely to modify it more than once or twice over the range of values selected 
for the willingness-to-pay. Provided we search for the correct number of decision changes for the 
parameter in question, the method calculates the true EVPPI values, asymptotically as the number 
of PSA samples goes to infinity. 

The algorithm developed by Sadatsafavi et al. is identical to that of Strong and Oakley de¬ 
scribed in the previous section in steps 1-4, but then proceeds as follows: 

5. Split the ordered list of net benefits into m = 1 .... ,M sub-lists C m . In this case, the list 
of known-distribution net benefit values is split at M points, (p Cl , ■ • ■, P Cm ), known as the 
segmentation vector. This means that the m —th sub-list contains all the net benefit values 
calculated with the value p such that p Cm < p < p c " m +' i . 

6. Calculate and store NB7 l (0) = j- ^ ' NBt(<p-'). 

jec m 

7. Maximise over all possible segmentation vectors to compute 

M 

EVPPI = max - V maxNB?’(6»). 

(<^i M ^ t 

m =1 

It is important to note that this method is also sensitive to the choice of M. It can be shown 
that the EVPPI estimate is asymptotically unbiased if we search for the true underlying number of 
decision changes. Therefore, Sadatsafavi et al. suggest a systematic method for choosing M, based 
on a visual tool for determining the number of decision changes. This tool plots the cumulative 
sum of the differences between the known-distribution net benefits for the two treatment options. 
When the optimal decision changes, so does the sign of the difference between the two treatment 
options. Thus, a change in optimal decision is shown by an extremum for this cumulative sum. 
An example of this visual tool is given in the supplementary material. Despite the usefulness of 
the visual tool, it can be difficult to identify extrema making EVPPI estimates using this method 
unreliable in certain situations. 

Both these single-parameter estimation methods are dependent on their inputs making “black¬ 
box” calculations difficult. In addition, since the publication of these two methods, further research 
has been developed that allows to calculate multi-parameter EVPPI using the PSA samples and 
non-parametric regression methods. 

3.3 Non-parametric regression for EVPPI calculations 

The basic idea proposed initially by Oakley [2J>| and developed further by Strong et al. [32] is 
that regression methods can be used to approximate the EVPPI. Specifically, Strong et al. (32 
demonstrate that the known-distribution net benefit can be approximated, based on sampled 
parameter values. In particular, since health economic models are likely to be complicated functions 
of the underlying parameters, flexible regression methods should be used, in order to limit the 
number of assumptions on the relationship between the known-distribution net benefit and the 
parameters. 

In a nutshell, for each simulated value of the parameters s, Strong et al. propose to approximate 
the inner conditional expectation in (JTJ as 

NB t (6E) = E*,*. [NB t (V> s , VO] + O 

with e s ~ Normal(0, a^). Furthermore, the conditional expectation can be thought of as a function 
of <fi only, as the conditional expectation is dependent on the value of cf>. Therefore, the problem 
can be formulated as 


NB t (<n = flt (vo+o 

for which the <fi s are “observed” values (obtained via Monte Carlo sampling) for the independent 
variables and NB t (0 s ) is the “observed” dependent variable. 

Strong et al. [32] provide R code [29] for two alternative non-parametric regression methods: 
Generalised Additive Models (GAMs) [T9] and Gaussian Processes [26]. Both these methods offer 
a large amount of flexibility but other regression methods could be used. In a general sense, the 
EVPPI estimate is calculated by: 

1. Sample S values (0 1 ,.... 0 s ) from the distribution of the parameters. 

2. Calculate the net benefit NBt(0 s ) for each observed parameter vector 6 s and treatment 
option t. 
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3. For each treatment option t, fit a regression curve with <fi a as the observed “covariates” 
and NB t (0 s ) as the observed “dependent” variable, for the s —th “unit” of analysis (i.e. 
simulations). 

4. For each treatment t, find the s —th fitted values gt{4> a ) by inputting the observed values (j) a 
into the regression curves. 

s s 

5. Calculate EVPPI = -i V'' ma xg t {<p a ) — max ^ V'' gt(<fi a ) 

iJ ‘ ■* t t O ‘ ■* 

s=l s=1 

This non-parametric regression method was published most recently and thus represents the cur¬ 
rent state of the research into estimation methods for the EVPPI. However, in spite of the relative 
complexity of this method, “black-box” calculations are possible using automatically chosen hyper¬ 
parameters for both non-parametric methods as they require little additional inputs from the user. 
This allows for the creation of general purpose functions (25j and the web application Sheffield 
Accelerated Value of Information (SAVI [30]) which allows users to calculate EVPPI. This web app 
also produces some standardised output, graphics, tables, that can be easily presented in formal 
reports. All these methods are also implemented in the R package BCEA m®. 


4 Comparison of methods 

This section compares the EVPPI estimates given by the methods discussed in this review and the 
two-step Monte Carlo procedure. Firstly, a case study pertaining to influenza infection vaccination 
is presented. We then discuss how the EVPPI estimates were calculated with regards to the 
input values for the single-parameter methods. Next, we present the results of our analysis; all 
five methods give approximately the same EVPPI estimate, with the same standard error for 
the four new methods. Therefore, the methods should be compared on their ease and speed of 
implementation and applicability. 

4.1 Case Study - Vaccine 

A Bayesian health-economic model is proposed to analyse the effect of the influenza vaccine on 
health-outcomes and costs. This example is relatively simple but complex enough to be intract¬ 
able analytically. Therefore, the parameters values must be sampled from their joint posterior 
distribution using Markov Chain Monte Carlo (MCMC) methods. 

Two treatment options are considered, either the vaccine is available to the population (t = 1), 
or not (t = 0). If a patient gets influenza, they are treated with anti-viral drugs and will often 
visit the GP. Complications may occur, including pneumonia and hospitalisation - in which case 
there will be some indirect costs such as time off work. The cost of the treatment is the acquisition 
cost of the drugs, the time in hospital, the GP visits and the cost of the vaccine. The benefit 
of the treatment is measured in QALYs (Quality Adjusted Life Years), where each adverse effect 
contributes negatively to the benefit. 

There are 28 key parameters in the model representing the probability of infection, the re¬ 
duction in risk due to the vaccine, the occurrence of complications, the monetary costs of the 
interventions and the QALY loss due to different health states. A detailed discussion of the full 
model parametrisation is presented in J2]. The single parameter EVPPI was calculated for all of 
these 28 parameters using all the methods outlined. 

4.2 Analysis 

This analysis compares the methods presented by estimating single parameter EVPPI, as some of 
the methods are only suitable for single-parameter estimation. As previously discussed, both single 
parameter estimation methods have input values that greatly affect the estimation. For the Strong 
and Oakley method, the number of subsets must be specified by the user and for the Sadatsafavi 
et al. method, the number of decision changes for each parameter is needed. The estimates are 
very sensitive to this choice. In fact, for the Sadatsafavi et al. method, if the visual tool indicates 
that the parameter does not affect the decision, this is equivalent to indicating that the EVPPI 
is equal to 0. This means that in some cases visual indications are used to determine the EVPPI 
rather than objective analysis. 

To determine these influential input values, the suggested visual tool was used for each of the 
28 parameters in the Vaccine case studjJ^] It was decided that 23 out of the 28 parameters change 

2 The supplementary material gives a more in depth explanation of this visual tool and implements it for some of the 
parameters 
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the optimal decision once, the other parameters have no effect - these correspond to an EVPPI 
equal to 0 in Table [T] For the Strong and Oakley m method, the upward bias of the EVPPI 
estimate was assessed for all the parameters in the data set. The number of subsets was taken 
as the largest number of subsets such that the upward bias remains below 0.1. The magnitude 
of the bias differed for each parameter, and therefore the number of subsets needed to reduce the 
upward bias below 0.1 was between 20 and 2. A larger number of subsets is associated with a 
larger estimate for the EVPPI and so it is important to assess the correct number of subsets for 
each parameter individually. 

For the two-step MC procedure the outer loop had 1000 observed posterior samples for all the 
parameters. As the case study is a Bayesian model, the inner loop of the 2 step MC procedure 
used MCMC methods to draw a sample from the joint posterior distribution for all the parameters. 
To ensure convergence and reduce autocorrelation, we took 2 chains of 100,000 samples each for 
the inner loop, with a burn-in of 9,500 and thinning of 500. Therefore, the eventual sample of the 
known-distribution net benefits had 1000 observations. The Monte Carlo error on this estimate 
is still quite large; ideally we would have sampled more observations in the inner loop. This was, 
however, too large a computational burden. To calculate the value of the current information, we 
sampled from our MCMC procedure 5,000,000 times, with a burn-in of 9,500 and then thinned to 
50,000. This sample size gives a Monte Carlo error on the estimate for the current information 
around 0.04. As only two treatment options are considered, the incremental net benefit was stored 
at the end of each loop. The maximum between the stored value and 0 was found for each loop. 

Finally, the two different non-parametric regression methods suggested by Strong et al. 
were used to calculate the EVPPI by approximating the inner expectation in (JT]). This analysis was 
carried out using the functions in |29j . For GAM regression, the functions use spline regression as 
smoothing functions. The R function gam is used by [29] to fit the spline regression and gam chooses 
the dimension of the underlying basis functions. Therefore, no additional input is needed for single 
parameter GAM regression. For Gaussian Processes, Strong et al. use a specific model structure 
to allow for the use of analytical results, therefore, no model structure needs to be specified by the 
user and functions are provided to calculate the EVPPI with no additional inputs. 

4.3 Results 

Table [l] displays the single parameter EVPPI estimates for the five different methods. All the 
methods give approximately the same results for each parameter. The two parameters with the 
highest EVPPI are the same across all five methods: /3i and wi. There is then some discrepancy 
in the ordering, but the results only differ slightly, certainly in comparison to the Monte Carlo 
error of the current information which is 0.04. 

Figure [l] shows the EVPI and the EVPPI for /3i and wi for different values of the willingness- 
to-pay, k. The analysis in this paper was undertaken for k = 20000 which is near the break-even 
point between the two treatment options. Both the EVPI and the EVPPI reach a peak when the 
optimal decision under current uncertainty changes. This is because the uncertainty is greatest at 
the point when both strategies are equal. As the EVPPI is very small for most of the parameters, 
even for a willingness-to-pay close to the break-even point, it is clear that uncertainty has little 
impact on the decision in this example. If the values of k were to change greatly in this example, 
we would be approximating very small EVPPI values and the MC error for the 2 step MC method 
would completely dominate the estimates. 

These results show that the approximation methods offer similar results for single parameter 
EVPPI estimates and can therefore all be used to estimate the EVPPI. The variation between 
the methods is likely to be due to the inherent nature of approximation. It is important to note 
that the accuracy of the estimate of the EVPPI should only be reported to 1 or 2 decimal places 
even when the analysis gives a much greater accuracy of calculation as all these methods only 
approximate the true EVPPI. 

Given that all five methods considered produce broadly similar estimates for the single para¬ 
meter EVPPI, it seems pertinent to investigate the standard error of the EVPPI calculated using 
these methods, as well as the computation time required to produce the estimate. The single¬ 
parameter estimation methods and the GAM regression take negligible time to find single para¬ 
meter EVPPI estimates. The GP regression is also fast for single parameter EVPPI, taking around 
2-4 seconds. The Monte Carlo procedure, however, takes around 55 minutes to calculate a single 
parameter EVPPI estimate. Therefore, all the approximation methods offer significant computa¬ 
tional savings. 

Taking these computational times into account, the four estimation methods are approximately 
equivalent, although the GP does take noticeably longer than the other methods. Therefore, an 
investigation of the standard error was undertaken to see if one method outperforms the others. 
Figure [5] shows the standard error for all the methods (excluding the Monte-Carlo procedure) for 
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Parameter 

Methods 

SO 

SAD 

GP 

GAM 

MC 

Probability of clinical outcomes 

A 

A 

A 

A 

A 

A 

A 

1.15 

0.26 

0.24 

0.14 

0.28 

0.27 

0.35 

1.17 

0.23 

0.15 

0.13 

0.25 

0.29 

0.4 

1.11 

0.11 

0.03 

0.08 

0.07 

0.18 

0.3 

1.11 

0.11 

0.03 

0.09 

0.12 

0.21 

0.3 

1.06 

0.25 

0.07 

0.05 

0.06 

0.21 

0.25 

Number of drugs prescribed 

<5 

0.3 

0.22 

0.11 

0.03 

0.06 

Probability of taking time off work 

V 

0.11 

0 

0.04 

0 

0.06 

Probability of receiving drugs 

71 

72 

0.19 

0 

0.17 

0 

0.11 

0.01 

0.11 

0.01 

0.06 

0.06 

Number of days off work 

A 

0.41 

0.44 

0.41 

0.43 

0.43 

Reduction in infection rate due to vaccine 

P 

0.3 

0.3 

0.28 

0.28 

0.28 

Probability of taking OTC medication 

z 

0.24 

0.13 

0.03 

0.03 

0.06 

QALY cost for each state 

V 1 

L04 

UJ5 

Ul 6 

UJ'j 

0.55 

0.2 

0.11 

0.18 

0.18 

0.57 

0.23 

0.17 

0.16 

0.29 

0.53 

0 

0.05 

0.12 

0.14 

0.53 

0 

0.05 

0.12 

0.16 

0.63 

0.06 

0.06 

0.13 

0.15 

Vaccine coverage rate 

<t> 

0.13 

0 

0.02 

0.02 

0.06 

Cost of resources 

A 

A 

A 

A 

A 

A 

A 

A 

0.17 

0.1 

0.37 

0.46 

0.44 

0.13 

0.2 

0.08 

0.17 

0.14 

0.37 

0.52 

0.38 

0.15 

0 

0 

0.13 

0.02 

0.31 

0.35 

0.24 

0.05 

0 

0 

0.13 

0.02 

0.31 

0.35 

0.38 

0.05 

0 

0 

0.07 

0.06 

0.29 

0.42 

0.25 

0.07 

0.07 

0.07 


Table 1: The EVPPI values for the parameters in the Vaccine example using 5 different methods 
discussed in this paper - Strong and Oakley single parameter estimation (SO), Sadatsafavi et al. single 
parameter estimation (SAL), Gaussian Process (GP) and GAM regression (GAM) and two-step Monte 
Carlo simulation (MC) 


the four parameters with the highest EVPPI. Clearly, the standard error is similar across all four 
methods and therefore there is little difference between the estimation properties of all 4 methods. 


5 Discussion 

This paper presents the current methods developed for approximating the EVPPI. It particularly 
focuses on four EVPPI estimation methods that have been developed since an earlier review 
paper on methods for estimating EVPPI | 13| . We compared the estimation properties of these 
4 estimation methods taking into the account the computational time required to approximate 
the EVVPI and the standard error of the estimate. Despite varying in complexity, these 4 new 
methods gave consistent results when compared with the 2-step MC procedure that derives directly 
from the definition of the EVPPI and can thus be considered to provide the “true” value. The 
four methods also have a similar standard error and computational time. Thus, their performance 
should be compared on the basis of simplicity of implementation. 

Two of the methods can only be used for single parameter EVPPI which clearly limits their 
applicability. Additionally, a thorough understanding of the methods and the respective tools for 
identifying the correct input values (visual tool, bias reduction) is essential before using of these 
methods. The BCEA [4] package in R allows users to calculate single parameter EVPPI using either 
method. However, the input values are still needed and a naive implementation of this function 
will lead the experimenter to grossly over/under-estimate the single parameter EVPPI. Therefore, 
these methods are largely superseded by the non-parametric regression method which are simpler 
to use as well as being applicable to multi-parameter EVPPI. 
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Expected Value of Perfect Partial Information 



Figure 1: The EVPI and EVVPI for f3\ and wi for willingness-to-pay thresholds varying between 0 
and 50000. The Sadatsafavi et al. method with one decision change is used to calculate the single¬ 
parameter EVPPI. 


As the GAM regression requires negligible computational effort for single-parameter EVPPI, 
it is the best method for approximating the EVPPI under these circumstances. However, for 
larger subsets, the interaction structure of the parameter must be specified by the user. Thus, 
the experimenter needs some understanding of GAM regression. Additionally, GAM regression is 
mathematically unstable for larger subsets of parameters (> 6). 

Therefore, the slower Gaussian Process regression is currently the only feasible option for larger 
parameter subsets, especially where the experimenter has no information about the interaction 
structures present in the parameter space. Unfortunately the Gaussian Process method takes 
more computational time as the number of parameters increases and there is a trade-off between 
the computational time spent fitting the model and the accuracy of the EVPPI estimate. For 
example, for a PSA data set of size N the computational effort for the Gaussian Process regression 
method is proportional to N 3 . Clearly, therefore, the smaller the PSA data set the faster the 
EVPPI calculation. However, the more data available to fit the non-parametric regression the more 
accurate the EVPPI estimate. An extension to the standard GP regression approach exploiting 
results from the geostatistics literature is being investigated [20] and results appear to be promising 
both in terms of accuracy and reduced computational time. 

All the methods investigated in this paper can be implemented in suitable software, such as R 
using the package BCEA [4] or using the standalone functions available from the original authors. 
Therefore, to some extent they can be applied in a “black-box” sense; which may lead to a more 
widespread use of EVPPI in health economic evaluations and allow decision makers to use Value 
of Information methods to inform their decisions. 


References 

[1] A. Ades, G. Lu, and K. Claxton. Expected Value of Sample Information Calculations in 
Medical Decision Modeling. Medical Decision Making , 24:207-227, 2004. 

[2] G. Baio. Bayesian methods in Health Economics. CRC Press, 2012. 

[3] G. Baio. BCEA: an R package to run Bayesian cost-effectiveness analysis. http://www. 
statistica.it/gianluca/BCEA/BCEA.pdf, 2012. 

[4] G. Baio, A. Berardi, and A. Heath. Bayesian Cost Effectiveness Analysis with the R package 
BCEA. Springer, forthcoming. 


9 



pi 


P7 


C\J 

i i , i 

C\J 



T T | J 



— GO 


— 00 


£ ° 


£ ° 


> 


> 


LD Tf 


LD ^ 

, i 

O 


O 

+ t | * 

O 


O 


O 


O 



SO SAD GAM GP SO SAD GAM GP 


\|/4 


0)1 


C\J 


C\J 






— 00 


— 00 


£ 


£ 


> 

■ i 

> 

1 1 Jc 

ID ^ 

j t 

ID ^ 

" 1 t T 

O 

T It 

O 

+ 

O 


O 


O n 


O 



SO SAD GAM GP SO SAD GAM GP 


Figure 2: The mean EVPPI estimate and its standard error for four estimation techniques - Strong 
and Oakley (SO), Sadatsafavi et al. (SAD), GAM and Gaussian Process (GP) regression - for the four 
most important parameters 


[5] A. Brennan and S. Kharroubi. Efficient Computation of Partial Expected Value of Sample 
Information Using Bayesian Approximation. Research Report 560/05, Department of Prob¬ 
ability and Statistics, University of Sheffield, UK, 2005. 

[6] A. Brennan, S. Kharroubi, A. O’Hagan, and J. Chilcott. Calculating Partial Expected Value 
of Perfect Information via Monte Carlo Sampling Algorithms. Medical Decision Making, 
27:448-470, 2007. 

[7] A. Briggs, M. Sculpher, and K. Claxton. Decision modelling for health economic evaluation. 
Oxford University Press, Oxford, UK, 2006. 

[8] J. Campbell, R. McQueen, A. Libby, D. Spackman, J. Carlson, and A. Briggs. Cost- 
effectiveness uncertainty analysis methods a comparison of one-way sensitivity, analysis of 
covariance, and expected value of partial perfect information. Medical Decision Making, page 
0272989X14556510, 2014. 

[9] Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evalu¬ 
ation of health technologies: Canada [3rd Edition]., 2006. 

[10] K. Claxton. Bayesian approaches to the value of information: implications for the regulation 
of new pharmaceutical. Health Economics, 8:269-274, 1999. 

[11] K. Claxton, P. Neumann, S. Araki, and M. Weinstein. Bayesian Value-Of-Information Ana¬ 
lysis. International Journal of Technology Assessment in Health Care, 17:38-55, 2001. 

[12] K. Claxton, M. Sculpher, C. McCabe, A. Briggs, R. Akehurst, M. Buxton, J. Brazier, and 
A. O’Hagan. Probabilistic sensitivity analysis for NICE technology assessment: not an op¬ 
tional extra. Health Economics, 14:339-347, 2005. 


10 






[13] D. Coyle and J. Oakley. Estimating the expected value of partial perfect information: a 
review of methods. The European Journal of Health Economics, 9(3):251-259, 2008. 

[14] Department of Health and Ageing. Guidelines for preparing submissions to the Pharmaceut¬ 
ical Benefits Advisory Committee: Version 4.3, 2008. 

[15] EUnetHTA. Methods for health economic evaluations: A guideline based on current practices 
in Europe - second draft, 29th September 2014. 

[16] J. Felli and G. Hazen. Sensitivity analysis and the expected value of perfect information. 
Medical Decision Making, 18:95-109, 1998. 

[17] J. Felli and G. Hazen. A Bayesian approach to sensitivity analysis. Health Economics, 8:263- 
268, 999. 

[18] E. Fenwick, S. Palmer, K. Claxton, M. Sculpher, K. Abrams, and A. Sutton. An Iterative 
Bayesian Approach to Health Technology Assessment: Application to a Policy of Preoperative 
Optimization for Patients Undergoing Major Elective Surgery. Medical Decision Making, 
26:480-496, 2006. 

[19] T. Hastie and R. Tibshirani. Generalized additive models, volume 43. CRC Press, 1990. 

[20] A. Heath, I. Manolopoulou, and G. Baio. Efficient High-Dimensional Gaussian Process Re¬ 
gression to calculate the Expected Value of Partial Perfect Information in Health Economic 
Evaluations. arXiv, 2015. 

[21] R. Howard. Information Value Theory. In IEEE Transactions on System Science and Cyber¬ 
netics, (1) 22-26, 1966. SCC-2. 

[22] J. Keisler, Z. Collier, E. Chu, N. Sinatra, and I. Linkov. Value of information analysis: the 
state of application. Environment Systems and Decisions, 34(l):3-23, 2014. 

[23] J. Madan, A. Ades, M. Price, K. Maitland, J. Jemutai, P. Revill, and N. Welton. Strategies for 
Efficient Computation of the Expected Value of Partial Perfect Information. Medical Decision 
Making, page 0272989X13514774, 2014. 

[24] D. Meltzer. Addressing uncertainty in medical cost-effectiveness analysis: Implications of 
expected utility maximization for methods to perform sensitivity analysis and the use of 
cost-effectiveness analysis to set priorities for medical research. Journal of Health Economics, 
2001 . 

[25] J. Oakley. Value of information for complex cost-effectiveness models. 2002. 

[26] C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. ISBN: 026218253X, 
2006. 

[27] M. Sadatsafavi, N. Bansback, Z. Zafari, M. Najafzadeh, and C. Marra. Need for speed: 
an efficient algorithm for calculation of single-parameter expected value of partial perfect 
information. Value in Health, 16(2):438-448, 2013. 

[28] L. Steuten, G. van de Wetering, K. Groothuis-Oudshoorn, and V. Retel. A Systematic and 
Critical Review of the Evolving Methods and Applications of Value of Information in Aca¬ 
demia and Practice. PharmacoEconomics, 31(l):25-48, 2013. 

[29] M. Strong. Partial EVPPI Functions. http://www.shef.ac.Uk/polopoly_fs/l.305039! 
/file/R_functions.txt, 2012. 

[30] M. Strong, P. Breeze, C. Thomas, and A. Brennan. SAVI - Sheffield Accelerated Value of 
Information, Release version 1.013 (2014-12-11), 2014. 

[31] M. Strong and J. Oakley. An efficient method for computing single-parameter partial expected 
value of perfect information. Medical Decision Making, 33(6):755-766, 2013. 

[32] M. Strong, J. Oakley, and A. Brennan. Estimating Multiparameter Partial Expected Value 
of Perfect Information from a Probabilistic Sensitivity Analysis Sample A Nonparametric 
Regression Approach. Medical Decision Making, 34(3):311-326, 2014. 

[33] E. Wilson. A Practical Guide to Value of Information Analysis. PharmacoEconomics, 
33(2):105-121, 2015. 


11 



A Visual Tool for Sadatsafavi et al. method 


The Sadatsafavi et al. m method for single parameter EVPPI estimation relies on an input value 
determining the number of changes in the optimal decision caused by the parameter of interest. To 
aid the choice of this value Sadatsafavi et al. suggest a visual tool that, for every pair of decisions 
t,t', can be used to determine the number of changes in optimal decision. Firstly we must order 
our PSA data in the parameter of interest, 0. Then, as a function of 0 we define 

s 

Ct.t'O) = | 5Z 7 (^ S < 0(NW) - NBt'On) (2) 

S = 1 

where NB t{4> 3 ) is the observed utility value for the s-th lowest value of 0 for treatment t. The 
extrema in this estimator will correspond to a change in optimal decision. For example, if t is the 
optimal decision until 0=1 then as 0 increases from 0 to 1 , only positive terms will be summed, 
then as the optimal decision changes after 1 , negative terms will be summed and therefore the 
value of C will decrease. 

We plot the observed values for C against the parameter of interest. This allows us to visually 
search for maxima or minima. As we only have empirical values and not a fully-specified function, 
a computer cannot find the extrema. Using the Vaccine case study, Figure [A] shows the visual tool 
for four the parameters to illustrate the difficulty faced in deciding whether a parameter affects 
the optimal decision. 

Firstly, we consider the tool for /3i (the top left-hand corner). Clearly, this plot has one 
extremum, around 0.06. There is a very obvious change in direction for the graph. This is the 
same for the graph for wi, where the direction change is clearly around 3.5. These two parameters 
are those with the highest EVPPI across all four methods, so the visual tool is highly effective for 
parameters with high EVPPI. 

Determining extrema is more challenging for /I 7 and p. The plot of rises steadily before 
possibly reaching a peak and decreasing slightly around 0.175. There may, therefore, be a decision 
change but is less obvious than for /3i and wi. For p it is challenging to ascertain whether we 
have an extremum at 0.625. There does seem to be some slight decrease to 0.0625 but it is very 
unclear. For this analysis, it was decided that p does cause the optimal decision to change giving 
an EVPPI of around 0.3. 
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